INTMDB-805: [Terraform] Create new TF Data Lake Pipelines Run data sources only (not resource) #1177
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ogs is set to true, when copying backups to other regions (#1150)
…elease-staging-v.1.10.0
…elease-staging-v.1.10.0
…elease-staging-v.1.10.0
…urces only (not resource)
andreaangiolillo
changed the title
themantissa
requested changes
May 22, 2023
| data "mongodbatlas_data_lake_pipeline_run" "test" { | ||
| project_id = "PROJECT ID" | ||
| name = "DATA LAKE PIPELINE NAME" | ||
| pipeline_run_id = "DATA LAKE PIPELINE RUN ID" |
There was a problem hiding this comment.
Could we help the user out here a bit more? For example, I think some will struggle to put together that they will first need to use data_lake_pipeline_runs for the name (which they will get from when they setup mongodbatlas_data_lake_pipeline) and that will give them the run id, e.g.:
resource "mongodbatlas_data_lake_pipeline" "pipeline" {
//assuming we've already setup project and cluster in another block
project_id = mongodbatlas_project.projectTest.project_id
name = "DataLakePipelineName"
sink {
type = "DLS"
partition_fields {
name = "access"
order = 0
}
}
source {
type = "ON_DEMAND_CPS"
cluster_name = mongodbatlas_cluster.clusterTest.name
database_name = "sample_airbnb"
collection_name = "listingsAndReviews"
}
transformations {
field = "test"
type = "EXCLUDE"
}
transformations {
field = "test22"
type = "EXCLUDE"
}
}
data "mongodbatlas_data_lake_pipeline_runs" "pipeline_run" {
project_id = mongodbatlas_project.projectTest.project_id
name = mongodbatlas_data_lake_pipeline.pipeline.name
}
data "mongodbatlas_data_lake_pipeline_run" "test" {
project_id = mongodbatlas_project.projectTest.project_id
name = mongodbatlas_data_lake_pipeline.pipeline.name
pipeline_run_id = mongodbatlas_data_lake_pipeline_runs.pipeline_run.pipeline_run_id
}
There was a problem hiding this comment.
I will add mongodbatlas_data_lake_pipeline to the example. However, based on my local testing, mongodbatlas_data_lake_pipeline won't return a run_id but the user has to go into the UI and manually trigger the pipeline run.
There was a problem hiding this comment.
@andreaangiolillo you bring up a good point about how this works. Since runs aren't something that really makes sense, from our scope review, to be managed by Terraform we've included them as a data source but not as a resource. But one can trigger an "On Demand Pipeline Run" via the UI, as you note, and API but unless a scheduled run or ondemand run are going you won't get a run_id back. So what I would suggest is to add "mongodbatlas_data_lake_pipeline_runs" to get the run_id but note in a comment something like pipeline_run_id will only be returned if a schedule or ondemand run is active FYI @Zuhairahmed this could be an area where our users will guide us if what we have offered is not sufficient.
There was a problem hiding this comment.
it makes sense. Thanks for the clarification. I will update the example
There was a problem hiding this comment.
Does mongodbatlas_data_lake_pipeline_runs make sense to include, if the user has to go to the UI and do the run to generate the id. At the point they're in the UI I expect it's just as easy for them to get the id and then provide that as a parameter for mongodbatlas_data_lake_pipeline_run
It feels like we're forcing Terraform to handle an interactive flow that is better in an interactive tool like the cli.
There was a problem hiding this comment.
@colm-quinn the user doesn't have to go through the UI, there is an API (though badly named in the docs so I put a ticket in to fix it https://jira.mongodb.org/browse/DOCSP-30114). Runs will give them all running runs so they can get the run_id if one exists. Really these data sources may not get used - I'm unsure. But when we've not included ones like this we get requests for them so we included in the scope. Sometimes we see Terraform used a bit like a config management tool so someone using a provisioner with the on demand run endpoint could want to get this data back. It's one of those fuzzy areas to me.
There was a problem hiding this comment.
Ah ok, so data sources are for our on-demand config/api style usage cases & resources for the terraform config & automation. Thanks for the details!
Zuhairahmed
approved these changes
May 22, 2023
There was a problem hiding this comment.
Few suggestions below, non-blocking. Thanks @andreaangiolillo and LGTM!
mongodbatlas/data_source_mongodbatlas_data_lake_pipeline_run.go
Outdated
Show resolved
Hide resolved
themantissa
requested changes
May 23, 2023
| data "mongodbatlas_data_lake_pipeline_run" "test" { | ||
| project_id = "PROJECT ID" | ||
| name = "DATA LAKE PIPELINE NAME" | ||
| pipeline_run_id = "DATA LAKE PIPELINE RUN ID" |
There was a problem hiding this comment.
@andreaangiolillo you bring up a good point about how this works. Since runs aren't something that really makes sense, from our scope review, to be managed by Terraform we've included them as a data source but not as a resource. But one can trigger an "On Demand Pipeline Run" via the UI, as you note, and API but unless a scheduled run or ondemand run are going you won't get a run_id back. So what I would suggest is to add "mongodbatlas_data_lake_pipeline_runs" to get the run_id but note in a comment something like pipeline_run_id will only be returned if a schedule or ondemand run is active FYI @Zuhairahmed this could be an area where our users will guide us if what we have offered is not sufficient.
Zuhairahmed
approved these changes
May 31, 2023
Zuhairahmed
approved these changes
May 31, 2023
|
Commenting to say this was reviewed by docs after merge and looks good 👍 |
maastha
pushed a commit
that referenced
this pull request
Jun 2, 2023
…urces only (not resource) (#1177)
martinstibbe
pushed a commit
that referenced
this pull request
Jun 10, 2023
…ta sources (#1173) * INTMDB-802: Add data_source and resource for Federated Database Query Limit * INTMDB-802: update provider.go * INTMDB-802: Add AtlasDataFederation Query Limit resource * INTMDB-802: Add AtlasDataFederation Query Limit resource * Update data_source_mongodbatlas_federated_query_limit.go * Update resource_mongodbatlas_federated_query_limit.go * INTMDB-802: temp * INTMDB-802: Add example * INTMDB-802: Add example * INTMDB-802: minor * INTMDB-802: example fix * Acceptance tests * Acceptance tests * Fix docs * Update main.tf * Update mongodbatlas/resource_mongodbatlas_federated_query_limit.go Co-authored-by: Andrea Angiolillo <[email protected]> * Update mongodbatlas/resource_mongodbatlas_federated_query_limit.go Co-authored-by: Andrea Angiolillo <[email protected]> * Update mongodbatlas/data_source_mongodbatlas_federated_query_limit.go Co-authored-by: Andrea Angiolillo <[email protected]> * Update mongodbatlas/resource_mongodbatlas_federated_query_limit.go Co-authored-by: Andrea Angiolillo <[email protected]> * Update mongodbatlas/resource_mongodbatlas_federated_query_limit.go Co-authored-by: Andrea Angiolillo <[email protected]> * Update website/docs/d/federated_query_limit.html.markdown Co-authored-by: Melissa Plunkett <[email protected]> * Update website/docs/d/federated_query_limit.html.markdown Co-authored-by: Melissa Plunkett <[email protected]> * Update website/docs/r/federated_query_limit.html.markdown Co-authored-by: Melissa Plunkett <[email protected]> * Update website/docs/r/federated_query_limit.html.markdown Co-authored-by: Melissa Plunkett <[email protected]> * Update website/docs/r/federated_query_limit.html.markdown Co-authored-by: Melissa Plunkett <[email protected]> * Update website/docs/d/federated_query_limits.html.markdown Co-authored-by: Melissa Plunkett <[email protected]> * Update website/docs/r/federated_query_limit.html.markdown Co-authored-by: Melissa Plunkett <[email protected]> * Update federated_query_limit.html.markdown * Update federated_query_limit.html.markdown * Address PR comments * Address PR comments * Update website/docs/d/federated_query_limit.html.markdown Co-authored-by: Zuhair Ahmed <[email protected]> * Update website/docs/d/federated_query_limits.html.markdown Co-authored-by: Zuhair Ahmed <[email protected]> * Update examples/federated-database-query-limit/README.md Co-authored-by: Zuhair Ahmed <[email protected]> * Update examples/federated-database-query-limit/README.md Co-authored-by: Zuhair Ahmed <[email protected]> * Update examples/federated-database-query-limit/README.md Co-authored-by: Zuhair Ahmed <[email protected]> * Update federated_query_limit.html.markdown * Update federated_query_limits.html.markdown * Update README.md * Address PR comments * Address PR comments * Apply suggestions from code review Co-authored-by: zach-carr <[email protected]> * INTMDB-805: [Terraform] Create new TF Data Lake Pipelines Run data sources only (not resource) (#1177) * INTMDB-803: [Terraform] Create a new Private Endpoint resource and data sources which supports Federated Database Instance and Online Archive (#1182) * Fix AWS provider version and example * updated federated instance * Addressed Comments * Update mongodbatlas/data_source_mongodbatlas_federated_query_limits.go Co-authored-by: Andrea Angiolillo <[email protected]> * Update data_source_mongodbatlas_federated_query_limits.go --------- Co-authored-by: Andrea Angiolillo <[email protected]> Co-authored-by: Melissa Plunkett <[email protected]> Co-authored-by: Zuhair Ahmed <[email protected]> Co-authored-by: zach-carr <[email protected]>
Zuhairahmed
added a commit
that referenced
this pull request
Jun 12, 2023
* fix: microsoft_teams_webhook_url keeps updating on every apply (#1148) * INTMDB-783: Point in Time Restore is not enabled when should_copy_oplogs is set to true, when copying backups to other regions (#1150) * Rebase v1.10.0 (#1156) * INTMDB-710: Serverless Instance wants to do an in-place update on every run (#1152) * INTMDB-694: Add PrivateEndpoint.srvShardOptimizedConnectionString to cluster (#1157) * INTMDB-780: analyzer argument in Atlas search index is required (issue #1132) (#1158) * INTMDB-809: upgrade atlas go-client to v0.26.0 (#1164) * INTMDB-809: upgrade atlas go-client * Update go.mod * remain doc updates * INTMDB-801: [Terraform] Create new TF Federated Database Instance resources and data sources (#1163) * INTMDB-408: Remove Deprecated Resources - cloud_provider resources, private_ip_mode, NEW_RELIC and FLOWDOCK in third_party_integration resource (#1159) * Remove deprecated resources * Deprecate additional related resource parameters * Add log message * Remove mongodbatlas_private_ip_mode * Add Full deprecation notice to Docs * more doc updates --------- Co-authored-by: Zuhair Ahmed <[email protected]> * INTMDB-466 - Added is_extended_storage_sizes_enabled (#1128) * Added is_extended_storage_sizes enabled * Fixed lint complaints * INTMDB-834: Address Follow Up comments in INTMDB-804 (#1181) * node_count docs update * INTMDB-804: [Terraform] Create new TF Data Lake Pipelines resource and data sources. (#1174) * INTMDB-805: [Terraform] Create new TF Data Lake Pipelines Run data sources only (not resource) (#1177) * INTMDB-803: [Terraform] Create a new Private Endpoint resource and data sources which supports Federated Database Instance and Online Archive (#1182) * INTMDB-806: Deprecate "mongodbatlas_data_lake" and "privatelink_endpoint_service_adl" (#1190) * Make aws field optional in federated_database_instance resource and upgrade aws provider (#1205) * INTMDB-835: Create resource mongodbatlas_cluster_outage_simulation (#1188) * Fix typos in docs for network peering resource imports (#1200) * Remove delete from create update (#1209) * INTMDB-781: [Terraform] Parameter Add: retainBackups in Cluster and Advanced_Cluster (#1210) * INTMDB-856: Add cloudProviderConfig to mongodbatlas_federated_database_instance (#1215) * INTMDB-655: PAK Resource Updates + Doc Cleanup + Deprecation Warnings (#1208) * Add project_assignment feature to permit assigning multiple projects to an API key * Add deprecation * lint * Add examples for project and org API key * Add additional multi project example * Update versions.tf * Add deprecation message create function to remove duplicated code * lint * Add new deprecation message * Bump terraform version * Update project_api_key.html.markdown * Update project.html.markdown * Update docs * Add validation of project_assignment parameter * Create README.md * Update README.md * Create PAK-upgrade-guide-1.10.0.html.markdown * Update PAK-upgrade-guide-1.10.0.html.markdown * Update project_api_key.html.markdown * Update website/docs/r/access_list_api_key.html.markdown Co-authored-by: Melissa Plunkett <[email protected]> * Update website/docs/d/access_list_api_key.html.markdown Co-authored-by: Melissa Plunkett <[email protected]> * Update examples/atlas-api-key/create-and-assign-pak-together/README.md Co-authored-by: zach-carr <[email protected]> * Update examples/atlas-api-key/create-api-key-assign-to-multiple-projects/README.md Co-authored-by: zach-carr <[email protected]> * Update examples/atlas-api-key/README.md Co-authored-by: zach-carr <[email protected]> * Update website/docs/r/project.html.markdown Co-authored-by: Zuhair Ahmed <[email protected]> * Update examples/atlas-api-key/create-and-assign-pak-seperately/README.md Co-authored-by: zach-carr <[email protected]> * Update examples/atlas-api-key/create-and-assign-pak-seperately/README.md Co-authored-by: Zuhair Ahmed <[email protected]> * Update examples/atlas-api-key/create-and-assign-pak-seperately/README.md Co-authored-by: zach-carr <[email protected]> * Add additional deprecation note * Update README.md * Bump up SDK to v0.29.0 * Update README.md More details to Readme explaining that all API keys are Organization Keys to help users --------- Co-authored-by: Zuhair Ahmed <[email protected]> Co-authored-by: Melissa Plunkett <[email protected]> Co-authored-by: zach-carr <[email protected]> * INTMDB-533: Feature Add: Programmatically Create Organizations (#1176) * Initial version * Acceptance testing issue for org creation * Add initial examples of org creation * skip tests for moment on organization resource * Add documentation * typo * terraform fmt * lint * go mod tidy * Documentation updates PR feedback * update TF version * Update variables.tf * Add additional Readme to show how to apply 2 step example * Update docs * Update examples/atlas-organization/Readme.md Co-authored-by: Zuhair Ahmed <[email protected]> * Update examples/atlas-organization/Readme.md * Update examples/atlas-organization/organization-step-1/Readme.md * Update examples/atlas-organization/organization-step-2/Readme.md Co-authored-by: Melissa Plunkett <[email protected]> * Update website/docs/r/organization.html.markdown * Update website/docs/d/organizations.html.markdown * Update go SDK add Org Update * Fix broken tests * Add isDeleted * Update go.sum --------- Co-authored-by: Zuhair Ahmed <[email protected]> Co-authored-by: Melissa Plunkett <[email protected]> * INTMDB-802: Create new TF Data Federation Query Limit resource and data sources (#1173) * INTMDB-802: Add data_source and resource for Federated Database Query Limit * INTMDB-802: update provider.go * INTMDB-802: Add AtlasDataFederation Query Limit resource * INTMDB-802: Add AtlasDataFederation Query Limit resource * Update data_source_mongodbatlas_federated_query_limit.go * Update resource_mongodbatlas_federated_query_limit.go * INTMDB-802: temp * INTMDB-802: Add example * INTMDB-802: Add example * INTMDB-802: minor * INTMDB-802: example fix * Acceptance tests * Acceptance tests * Fix docs * Update main.tf * Update mongodbatlas/resource_mongodbatlas_federated_query_limit.go Co-authored-by: Andrea Angiolillo <[email protected]> * Update mongodbatlas/resource_mongodbatlas_federated_query_limit.go Co-authored-by: Andrea Angiolillo <[email protected]> * Update mongodbatlas/data_source_mongodbatlas_federated_query_limit.go Co-authored-by: Andrea Angiolillo <[email protected]> * Update mongodbatlas/resource_mongodbatlas_federated_query_limit.go Co-authored-by: Andrea Angiolillo <[email protected]> * Update mongodbatlas/resource_mongodbatlas_federated_query_limit.go Co-authored-by: Andrea Angiolillo <[email protected]> * Update website/docs/d/federated_query_limit.html.markdown Co-authored-by: Melissa Plunkett <[email protected]> * Update website/docs/d/federated_query_limit.html.markdown Co-authored-by: Melissa Plunkett <[email protected]> * Update website/docs/r/federated_query_limit.html.markdown Co-authored-by: Melissa Plunkett <[email protected]> * Update website/docs/r/federated_query_limit.html.markdown Co-authored-by: Melissa Plunkett <[email protected]> * Update website/docs/r/federated_query_limit.html.markdown Co-authored-by: Melissa Plunkett <[email protected]> * Update website/docs/d/federated_query_limits.html.markdown Co-authored-by: Melissa Plunkett <[email protected]> * Update website/docs/r/federated_query_limit.html.markdown Co-authored-by: Melissa Plunkett <[email protected]> * Update federated_query_limit.html.markdown * Update federated_query_limit.html.markdown * Address PR comments * Address PR comments * Update website/docs/d/federated_query_limit.html.markdown Co-authored-by: Zuhair Ahmed <[email protected]> * Update website/docs/d/federated_query_limits.html.markdown Co-authored-by: Zuhair Ahmed <[email protected]> * Update examples/federated-database-query-limit/README.md Co-authored-by: Zuhair Ahmed <[email protected]> * Update examples/federated-database-query-limit/README.md Co-authored-by: Zuhair Ahmed <[email protected]> * Update examples/federated-database-query-limit/README.md Co-authored-by: Zuhair Ahmed <[email protected]> * Update federated_query_limit.html.markdown * Update federated_query_limits.html.markdown * Update README.md * Address PR comments * Address PR comments * Apply suggestions from code review Co-authored-by: zach-carr <[email protected]> * INTMDB-805: [Terraform] Create new TF Data Lake Pipelines Run data sources only (not resource) (#1177) * INTMDB-803: [Terraform] Create a new Private Endpoint resource and data sources which supports Federated Database Instance and Online Archive (#1182) * Fix AWS provider version and example * updated federated instance * Addressed Comments * Update mongodbatlas/data_source_mongodbatlas_federated_query_limits.go Co-authored-by: Andrea Angiolillo <[email protected]> * Update data_source_mongodbatlas_federated_query_limits.go --------- Co-authored-by: Andrea Angiolillo <[email protected]> Co-authored-by: Melissa Plunkett <[email protected]> Co-authored-by: Zuhair Ahmed <[email protected]> Co-authored-by: zach-carr <[email protected]> * Update .github_changelog_generator * Update CHANGELOG.md --------- Co-authored-by: maastha <[email protected]> Co-authored-by: Andrea Angiolillo <[email protected]> Co-authored-by: Zuhair Ahmed <[email protected]> Co-authored-by: Dosty Everts <[email protected]> Co-authored-by: Johanna Öjeling <[email protected]> Co-authored-by: Melissa Plunkett <[email protected]> Co-authored-by: zach-carr <[email protected]>
andreaangiolillo
added a commit
that referenced
this pull request
Jun 12, 2023
…urces only (not resource) (#1177)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Ticket: INTMDB-805
Type of change:
Required Checklist:
Further comments
Acceptance Tests
Running tool: /Users/andrea.angiolillo/.asdf/shims/go test -timeout 300000s -run ^TestAccBackupDSDataLakePipelineRun_basic$ github.com/mongodb/terraform-provider-mongodbatlas/mongodbatlas ok github.com/mongodb/terraform-provider-mongodbatlas/mongodbatlas 0.557sRunning tool: /Users/andrea.angiolillo/.asdf/shims/go test -timeout 300000s -run ^TestAccBackupDSDataLakePipelineRuns_basic$ github.com/mongodb/terraform-provider-mongodbatlas/mongodbatlas ok github.com/mongodb/terraform-provider-mongodbatlas/mongodbatlas (cached)